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ABSTRACT 

A "meta-assessment" was done of 13 pilot projects on 
student outcomes assessment in a variety of disciplines at 11 
campuses in the California State University (CSU) system* These 
projects had developed both quantitative and qualitative strategies 
for collecting data on student learning outcomes. The meta-assessment 
was designed to identify key factors shared by successful projects. A 
three-part framework was used to specify and organize variables by 
which the projects were evaluated. Environmental and methodological 
factors served as "predictor" variables and project outcomes served 
as "criterion" scores. General factors that contributed to the 
relative effectiveness of the projects were identified by assessing 
the relationship between "predictor" and "criterion" variables. The 
assessment found that recruiting and maintaining faculty support was 
a key variable in project success but also showed high variability 
across projects. Administrative support was consistently reported as 
important, and better integration of the assessment agenda in 
campus-level policy and more concrete recognition of assessment 
activities for professional development was recommended. Project 
directors' training and experience in measurement and analysis was 
also identified as important. The key methodological variable was the 
development or adoption of adequate measures of student outcomes. 
(JB) 
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Introduction 



The current assessment movement in higher education is driven by the wary 
partnership of reform and accountability, a partnership yielding a complex and 
diverse collection of assessment activities in university settings (Ewell, 1991). 
Over the last several years, the California State University (CSU) system has 
moved to construct an assessment agenda that responds to both reform and 
accountability in ways that preserve the commitment of the CSU to intellectual 
and programmatic diversity. Beginning in 1986, the Academic Program 
Improvement (API) Campus Grants Program supported a series of student 
outcomes assessment projects initiated by faculty in a variety of disciplines on 
eleven campuses of the CSU. These projects developed both quantitative and 
qualitative strategies for collecting data on student learning outcomes. Data 
from these initiatives provided the basis for planning ongoing programs of 
assessment in General Education and baccalaureate degree programs 
throughout the CSU. 
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Background: 

The Evaluation Study 

of CSU Pilot Projects 



An evaluation study, supported by a grant from the U.S. Department of 
Education Fund for the Improvement of Postsecondary Education (FIPSE) and 
conducted under the aegis of the CSU Institute for Teaching and Learning, 
provided a "meta-assessment" of the thirteen pilot projects. The pilot projects, 
briefly described in Table 1 , developed and field-tested a diverse set of 
assessment measures, including portfolios, interviews, senior/capstone projects, 
surveys, and examinations. The "meta-assessment" of the projects was 
designed to identify key factors shared by successful projects. The pilots thus 
made it possible to define elements of effective assessments in the CSU and to 
disseminate the best practices developed by these experimental projects. 
Moreover, the diversity of the projects, both disciplinary and curricular, 
provided information about the essentials of good assessment practice across 
curricular boundaries. 
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List of API Student 
Outcomes Assessment 
Projects 



Table 1: CSU Student Outcomes Assessment Pilot Projects 

1986-1989 



Director/Campus 

Andrew Moss 
Pomona 



Grant Year 

$39,374 1986 



Kenneth Nyberg 
Bakersfield 



$66,708 1986,87 



Priscilla Chaff e-Stengel 
Fresno 



$22,562 1987 



Project Title & Focus 

Enhancing Quality by Assessment: 
A General Education Project 

"ocl's; Development of a comprehensive 
assessment program for an interdisciplinary 
General Education program 

An Empirical Evaluation 

of Five Baccalaureate Social Science 

Programs 

Focus: Development of a model to 
conduct longitudinal assessments or student 
performance and perceptions of degree 
programs in anthropology, economics, 
political science, psychology, and sociology 

Assessment of Undergraduate 
Writing Competence 

Fccls: Assessment of student performance 
on the Upper Division Writing Exam as a 
function ot course exposure and language 
proficiency 
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Table 1 — Continued 



Director/Campus 

Catherine Dezseran 
Peter Grego 
Northridge 



Grant Year 

$25,532 1987 



Project Title & Focus 

Student Outcomes Assessment in 
Academic Program Improvement in 
Theatre 

Focus: Development of a performance- 
based mastery test for summative <ind 
fo r mative assessment of student achieve- 
ment in theatre 



Leigh Mintz 
Lu Ann Duffus 
Hayward 
Richard Giardina 
Newman Fisher 
San Francisco 
Leon Dorosz 
Howard Sheilharnmer 
San Jose 

Harry Polkinhorn 
San Diego 



$ 9,880 1987 



$10,978 



$10,978 



$25,467 1987 



Prisciiia Chaffe-Stengel $27,490 1 988 
Fresno 



S. Eugene Clark 
Bakersfield 



$27,125 1988 



Assessment of Majors: A Three-Campus, 
Three-Discipline Model 

Focus: Development of comprehensive 
examinations for seniors in biology, 
economics, and mathematics 



Student Outcomes Assessment: 
Liberal Studies Major 

Focus: Development of a multi-measure 
assessment program for student outcomes 
in liberal studies 

Assessment of Undergraduate Reading 
Competence 

Focus: Assessment of student reading 
strategies and competence related to course 
assignments and library skills 

Knowledge and Attitudes in General 
Education: A CSU-Community College 
Joint Assessment 

Focus: Assessment of impact of GE course 
in Western civilization on students' 
knowledge and values 



8 



Table 1 — Continued 



Director/Campus 

P. Chris Cozbv 
Jeffry Young 
Fullerton 



Catharine Lucas 
San Francisco 



Mary I u Mattson 
Sonoma 



Mary Cuiiinan 
Havward 



Bessie Marquis 
Chi co 



Grant Year 

$27,500 1988 



$18,692 1988 



$49,172 1988,89 



$20,635 1989 



$28,366 1989 



Project Title & Focus 

Student Outcomes Related to 
Curricular Variety in Gerontology 

z ccjs: Development of a model for cross 
campus assessment of outcomes tor 
interdisciplinary programs in gerontology 

Assessing Outcomes for English 
Teacher Candidates 

z ocjs\ Development of an assessment 
course to evaluate the subject-matter 
competency of teacher credential candi- 
dates in English language arts 

Integrating Student Outcomes Assessment 
into the Curriculum 

Focus: Development ot a portfolio system 
to assess formative and summative 
outcomes for students in an interdisciplinary 
liberal studies program 

Assessment of Student Outcomes: A Basic 
Writer's Writing Program 

-ecus: Development of a model for 
assessing outcomes in the Intensive 
Learning Experience writing course 
sequence 

Outcomes Assessment of Four Classes of 
Nursing Graduates 

-ccls: Development of a multi-measure 
assessment of nursing program graduates to 
identify trends in program effectiveness 
from 1983-89 



Summary of Campus 
Projects: 1986-1989 



The diversity of the thirteen projects studied is apparent in their different 
settings, focus, and strategies. They shared, however, a single goal: the creation 
of models for assessment with application to comparable programs throughout 
the CSU. Project directors addressed this goal at four different levels: 
measurement, program, campus, and course. 



[The projects] sharea\..a single goal: the creation ofmodehfor assessment 
with application to comparable programs throughout the CSU. 



ERIC 



At the measurement level, an assessment of theatre program outcomes at CSU 
Northridge, directed by Professors Catherine Dezseran and Peter Grego, was 
part of a departmental self-study. The project was aimed at ccrricular revision 
through the development of a mastery test assessing theatre knowledge, plus an 
evaluation of theatre ensemble teaching methods. One innovative result was 
the development of a faculty-generated Performance Assessment Instrument tor 
measuring students' achievement in production areas of theatre. An assessment 
project in English language arts undertaken by Professor Catharine Lucas at San 
Francisco State University explored the use of an "assessment cour:e" to certify 
the subject matter competency of teacher credential candidates. The 
assessment course, an alternative to standardized tests (inadequate) or direct 
observations (too costly), evaluated candidates' language arts preparation, in 
four areas: oral, written, response to literature, and applied concepts. A third 
measurement-level project piloted an exit assessment for majors in 
mathematics/biology/economics. Project directors at three "sister" CSUs 
(Professors Leigh Mintz and Lu Ann Duffus at CSU Hayward, Professors 
Richard Giardina and Newman Fisher at San Francisco j'.ate University, and 
Professors Leon Dorosz and Howard Sheilhammer at San Jose State University) 
worked with faculty in the three majors to design major "comp" exams The 
assessment used a locally designed instrument administered by faculty from 
cooperating departments in this three-campus consortium. 

In an ambitious program-level project, Professors P. Chris Cozby and Jeffry 
Young at CSU Fullerton developed a systemwide, interdisciplinary assessment 
of all programs in the CSU with course content in gerontology. One of their 
goals in describing CSU gerontology offerings was to reconcile program 
objectives with curriculum planning and assessment activities in gerontology. 
In addition, the survey developed in this project was intended as a model for 
continuing'assessment of academic/career training in gerontology for current 
students and alumni. Professor Marylu Mattson at Sonoma State University 
designed a portfolio assessment system as an alternative to traditional 
outcomes measures for the Hutchings Interdisciplinary General Education 
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program. Recognizing that a persistent issue for portfolio assessments is faculty 
ownership, Mattson was careful to make faculty input a central feature of this 
experimental program-level assessment. At CSU Chico, Professor Bessie 
Marquis directed an assessment of Nursing School program outcomes, 
evaluating competency test outcomes, workplace competency, and the relation 
of program performance indicators (e.g., GPA) to job performance. English 
Professor Andrew Moss conducted an extensive program evaluation in the 
Interdisciplinary General Education Program at Cal Poly Pomc:;a. In 
consultation with Peter Ewell, course-related assessment teams worked to 
produce guidelines for using assessment results as a tool for curricular and 
instructional improvement at Cal Poly. An evaluation of student learning 
outcomes in five majors in the CSU Bakersfield School of Social Sciences was 
directed by sociology Professor Kenneth Nyberg. This program-level initiative 
was adapted in part from models suggested by assessment programs in 
Tennessee and Missouri. The goal ot the Nyberg study was a longitudinal 
description of learning outcomes and perceptions of undergraduate majors in 
anthropology, economics, political science, psychology, and sociology. 
Finally, a multi-measure assessment of student outcomes in the liberal studies 
program at San Diego State Universitv was directed by English Professor. Harry 
Poikmnorn. This pilot assessment usea data from the ACT 'Coilege Outcomes 
Measures Program" as a starting'point for program evaluation. 

Both campus-level projects in the pilot group were conducted by Professor 
Priscilla Chaffe-Stengel at CSU Fresno. Aimed at the universal CSU writing and 
reading competency requirements, these projects used multiple indices to look 
at factors related to students' writing performance (measured by the Upper 
Division Writing Exam), and reading proficiency and behaviors (measured by 
standardized tests as well as by library use patterns, etc.). One of Chaffe- 
Stengei's goals for this assessment was the creation of "predictive profiles" for 
use in advisement. 

At the course level, an assessment directed by Professor S. Eugene Clark at 
CSU Bakersfield measured pre- and post-course knowledge and attitudes in 
General Education courses in Western civilization/European history. An 
important focus of this project was to examine the "equivalency'' of courses 
offered at Bakersfield College and CSU Bakersfield. As the project director 
noted, a common core was assumed in articulation agreements but faculty 
rarely had the opportunity to "examine the equivalent courses and student 
outcomes." A second course-level project, directed by Professor Marv 
Cullinan at CSU Hayward, addressed learning outcomes in remedial writing 
through assessment of student outcomes in Hayward's Intensive Learning 
Experience (ILE) writing course sequence. Noting the shift toward greater 
reliance on the remedial writing course program on this campus, director 
Cullinan stated that the project specifically addressed the need to collect 
information on writing outcomes for non-native speakers, who comor^ed -0 
percent of CSU Hayward's basic writing students. 




Evaluation 



A three-part framework was used to specify and organize variables by which 
the projects were evaluated. The framework is illustrated in Table 2. 
Environmental and methodological factors served as "predictor" variables. 
Project outcomes served as "criterion" scores. General factors that contributed 
to the relative effectiveness of the projects were identified by assessing the 
relationship between "predictor" and "criterion" variables. For a detailed 
description of this analysis and results, refer to Riggs & Worthiey, 1992, 

ANALYSIS OF ENVIRONMENTAL FACTORS 
Faculty Involvement 

Projects differed significantly in reference to the inclusion of faculty 
participants in the planning/deveiopment of the project, faculty agreement with 
the nature of student performance criterion, perceived workload resulting from 
the project, and perceptions of ownership among faculty. 



Faculty participation in project implementation and the perceived 
faculty workload were important indicators of assessment success. 



The number of faculty involved in the initial planning of each project ranged 
from 2 to 1 DO with a mode of 4. The number of faculty involved in project 
implementation ranged from 2 to 400 (one project ended up universitywide in 
its second year) with a mode of 14. Directors generally reported high levels of 
faculty participation in the implementation of the project, support for the 
subsequent goals of the project, and consensus with the project plan. There 
were, however, some glaring exceptions. One director simply reported that 
"lack of faculty support made implementation impossible." 

Faculty participation in project implementation and the perceived faculty 
workload were important indicators of assessment success. Faculty ownership 
(the perception that the project was locally controlled) and general faculty 
consensus with the assessment plan were also relatively good indicators. 
Participation of all faculty in initial planning stages appeared to have little to do 
with project outcomes. 

Training/Experience of Faculty Participants 

Project directors were almost exclusively senior/tenured members of their 
departments. The directors were also generally wc!l-read and relatively 
experienced with student outcomes assessment. There was more variance in 
reference to other participating faculty. Many participants were hearing of 
''student outcomes assessment" for the first tirrife.O 



Table 2: Factors Evaluated in Relation to Project Outcomes 



Environmental Factors 



Methodological Factors 



Faculty Involvement 

Planning Participation 
Faculty Participation in Project 
Faculty Ownership 
Consensus with Plan 
Perceived Faculty Workload 



Training/Experience 

Director's Training 
Faculty Training 

Director's Academic Experience 

Support Variables 

Budget and Supplies 
Administrative Support 
Student Support 

Existing Procedures 

Previous Assessment 

Project Focus 

Content Domain 
Breadth of Audience 



General Procedural Adequacy 

Goal Definition 
Selection of Outcomes 
Measures Developed 
Data Collection/Reporting 
Measurement Properties 
Statistical Analysis 

Project Comprehensiveness 

Multicultural Issues 

Development of Multiple Measures 

Report Comprehensiveness 

Cost Effectiveness 

Utility/Economy 



Outcomes 



Direct Outcomes 

Project Continuing 
Additional Funding 
Gains in Student Achievement 
Curricular Impact 
Better Teaching 
Feedback to Students 
New Measures Developed 
Better Data Use 
Dissemination of Results 



Indirect Outcomes 

Recruitment/Retention 
Attitudes Toward Assessment 
New Moneys for Assessment 
Visibility of Assessment 
External Adoption 
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Of the three factors describing the training and background of project 
participants, only the project director's training and experience specific to 
student outcomes assessment ranked very high in association with project 
outcomes. Adequate planning and guidance by the project directors appeared 
sufficient to offset any lack of experience ~* project participants. 

Support Variables , 

Significant variance existed in reference to project support variables such as 
budget and supplies. Ten directors reported receiving additional funds beyond 
the required matching funds from their university. 

Administrative support was generally strong. No projects reported direct 
administrative resistance to student outcomes assessment, although there were 
varying degrees of administrative "red tape" associated with project 
implementation. 

Student support was more mixed. Directors reported that students generally 
supported the goals of the projects, but frequently offered resistance to changes 
associated with the implementation of outcomes assessment. Resistance was 
especially prevalent when additional coursework and/or "tests" were part of 
the plan. Apparently, students were as sensitive to the "workload" issue as 
were faculty members 

As for the subsequent impact of support variables on project outcomes, 
administrative support was most clearly tied to project success. Though a goal 
of the CSU system is to establish faculty-initiated outcomes assessment, these 
initiatives will most likely fail without the support of academic administrators. 
University administration sets the "tone" in reference to. the academic 
legitimacy of such efforts. Unless outcomes assessment efforts are rewarded 
and supported by administration, it will be difficult to maintain faculty 
motivation. 

Student support appeared only moderately related to reported outcomes. 
Though this source of support was assessed through the eyes of the project 
directors rather than from the students themselves, this result dues not appear 
unreasonable. A few directors did speak to the fact that students will usually 
support any program modifications properly justified by the faculty. 



Unless outcomes assessment efforts are rewarded and supported by admin- 
istration, it will be difficult to maintain faculty motivation. 



Support in the form of budget and supplies was unrelated to project success. 
A probable post hoc explanation is that many of the directors were supporting 
their project "out of their hide." The efforts of some directors clearly went 
beyond what might have been expected given the modest budgets they were 
receiving for their administration of each project. This phenomenon was more 
likely to occur if the director was working in a content area for which 
outcomes assessment research could be considered legitimate professional 
development or for senior faculty members who had achieved tenure. 



Another possible explanation for the inconsistency between budget and project 
success would be the inherent differences in requirements for adequate 
assessment procedures across different content areas. It may be possible that 
adequate assessment can be achieved for relatively small costs in some 
disciplines, but will be very expensive in others. Consequently^adequate 
results might be obtained for some even when resources are tight while others 
will find assessment prohibitive without additional budgetary support. 

Existing Student Outcomes Assessment Procedures 

Seven directors reported existing outcomes assessment programs in place at the 
time of their pilot projects. When asked to describe how existing procedures or 
attitudes might have contributed to the effectiveness of the pilot projects, 
responses were diverse. Five directors reported negative impact. Faculty 
suspicion and apathy were the major hurdles reported. One director reported 
that the faculty saw the project "... as a threat at worst and a waste of time at 
best." 

There were four reports of positive impact. Faculty education and exposure to 
successful student outcomes assessment procedures was reported to have been 
helpful to two of the projects. An existing faculty desire to clarify program goals 
was reported as a contributing factor to the success of one project. 

While these reports were somewhat mixed in the final analysis, project 
outcomes showed very little association with these perceived existing attitudes. 
The good news of this result is that projects breaking new ground do not 
necessarily have to anticipate damaging levels of resistance. The bad news may 
be that previous experience with outcomes assessment may not guarantee that 
new initiatives will be welcomed with open arms. 

Project Focus 

A quick review of Table 1 confirms the fact that the content areas of projects 
varied considerably. Intended "breadth of audience" also varied. Nine directors 
indicated that they had shared results beyond the reports requested by funding 
sources. Six projects have achieved nationwide recognition via national 
organizations, conferences, or major publications. 

The institutional level at which projects were implemented filled the entire 
range from a single selected class to a project that is now replicating the 
project at eighteen campuses nationwide. Altogether, three were applied to 
selected classes, four were departmentwide, two were at the school level, two 
were universitywide, and two were at least systemwide. 



The content area in which the project occurred had little to do with the 
relative success of the project. 



The content area in which the project occurred had little to do with the relative 
success of the project. However, the intended breadth of the audience iwho 
and how many individuals/organizations the project director expected to learn 
about the results of his/her project) was very closely associated with outcomes, 
especially indirect outcomes. 




While it might have been expected that content areas such as education or 
behavioral sciences would be more readily adapted to outcomes assessment, 
findings of the evaluation study did not support this conclusion. Projects in 
humanities and sciences were equally as successful as those in education and 
behavioral sciences. This is good news for campuses that hope to initiate 
outcomes assessment across a universe of content domains. 

The tight association between the intended breadth of the audience for the 
project's results and project outcomes is somewhat surprising, but is perhaps 
logical in that the excellence of project outcomes should be related to how 
many individuals or organizations with which the director intends to share the 
results. This result may also reflect the director's experience and enthusiasm for 
student outcomes assessment. As previously observed, project director efficacy 
in outcomes assessment was closely associated with project outcomes. 

ANALYSIS OF METHODOLOGICAL FACTORS 

General Procedural Adequacy 

Project goals were as diverse as the programs they were intended to serve, and 
are consequently difficult to summarize. Many had the objective of testing 
feasibility of outcomes assessment within the specific academic program. 
Another common objective was to try out different types of external evaluators. 
The most common objective not directly related to the assessment process itself 
was the desire to clarify/develop programmatic objectives. 

The types of outcomes assessed also varied dramatically. They included: 
assessment of simple content knowledge; demonstration of specific process 
skills; student, alumni, faculty, and the public's attitudes toward the program; 
attainment of post-graduate goals (e.g., employment status, general satisfaction 
with preparation, employers' satisfaction with the program's graduates); and 
the development of specified attitudes/beliefs. 

Many methods of outcome measurement were employed. These included 
written examinations (objective and essay), oral examinations, personal 
interviews, graded assignments, project evaluations, and attitudinal rating 
scales/surveys. 



The most common objective not directly related to the assessment process 
itself was the desire to clarify/develop programmatic objectives. 



In reference to general procedural adequacy, the most important aspect of the 
assessment projects appeared to be the adequacy of the measures they 
developed or adopted. It is highly logical to expect that project success would 
hinge on the ability of the measures used to reliably and validly measure 
student outcomes. This process begins with the selection of appropriate 
outcomes to measure. 

Other variables in this category, though not highly ranked in their association 
with direct outcomes, were toward the top of the list in relation to indirect 
outcomes. Much of this relationship appeared dependent upon the association 
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with the indirect outcome of external adoption. Those projects bound for 
adoption were generally the most precise in the definition of project goals and 
most ambitious and accurate in the production of statistical analyses. 

Project Comprehensiveness 

The projects varied somewhat in how many methods of evaluation were used 
by each project. At least four used only r single type of measure. Projects 
varied dramatically in their focus on sensitivity to multicultural issues. 
Multicultural issues were integral to the goals of some projects while others did 
not even collect appropriate demographic information. The "report 
comprehensiveness" of projects also varied dramatically. While some provided 
highly detailed volumes of documentation of their activities, others barely 
completed summative reports to meet minimum requirements of their funding 
agencies. 

The "development of multiple measures" of student outcomes and "sensitivity 
to multicultural issues" appeared moderately associated with project outcomes. 
The comprehensiveness of the reports made available was not closely related 
to project outcomes. 

The relatively high ranking of "sensitivity to multicultural issues" and the 
"development of multiple measures" may again reflect the sophistication of the 
project director in reference to good outcomes assessment. It is also reasonable 
to expect projects that developed or used more than one form of assessment to 
be more successful. 

Sensitivity to multicultural issues would appear especially relevant to the 
potential for recruitment and retention of underrepresented students. This 
would partially explain the relatively high ranking of this variable in reference 
to indirect outcomes. 



Sensitivity to multicultural issues would appear especially relevant to 
the potential for recruitment and retention of underrepresented 
students. 



The low association between the comprehensiveness of project reports and 
project outcomes might indicate that much occurred but was not completely 
reported. Many directors indicated that deadlines fell before they had time to 
adequately process the project results. Some compensated by disseminating 
results via other channels (e.g., regional and national presentations, journal 
publications). 

Cost Effectiveness of Project 

While projects varied dramatically in their cost per student participant, the 
"utility/economy" of the projects' assessment procedures was near the bottom 
of the rank ordering in reference to its association with project outcomes. 
Apparently expensive projects in terms of their dollars-spent-to-students- 
assessed were not always the richest in results. 

This result is perfectly reasonable given the experimental nature of these 
projects. Directors tried a wide variety of asses^ient procedures to see which 
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might work best. Also, there may be some cost differences across disciplines 
necessitated by the different nature of assessment processes. Some content 
areas may be able to conduct excellent "cheap" assessment, while other areas 
may only achieve moderate results despite a high price tag. 

Nevertheless, there is some logic to the assumption that efficient assessment 
will produce more desirable results. As more data is collected, this relationship 
should be reassessed within content areas. 

PROJECT OUTCOMES 

Direct Outcomes 

A critical outcome was defined as continuation of the project beyond the 
original year of funding. Seven projects were not continued. Lack of funding 
was the main reason given for program closure. Of those projects that 
continued, some had obtained no additional funding, and some were receiving 
very minimal departmental/university support. The projects that reported 
receiving additional support included one with an API continuation grant, one 
with an AAC/FIPSE grant orginating before the API grant, and one with no 
report of its funding source. 

Directors were also asked to assess the level of achievement of different 
potential project outcomes. Of those outcomes identified as direct outcomes, 
almost all directors reported development of good measures of student 
outcomes as an achievement of their projects. Most reported curricular 
improvement, increase in student feedback, clarification of instructional goals/ 
objectives, increase in faculty assessment skills, and successful dissemination 
of information to other departments/schools/universities. As a whole, d'rectors 
were less certain about gains in student achievement, improvement in teaching 
by the faculty involved, and development of new or improved uses of existing 
databases. 

Indirect Outcomes 

Of those outcomes identified as indirect outcomes, most directors reportea 
improved self-evaluation of the academic program, improved faculty attitude 
toward assessment activities, and greater visibility of assessment activities. 
Overail, somewhat less impact was perceived on student recruitment. ?tucert 
attitude toward assessment activities, and institutional attitude toward 
assessment. Directors reported less success with the development of new 
sources of revenue/support. Seven directors reDorted that thev were able to 
successfully adopt or use methods developed by other departments or 
institutions. 



Of those outcomes identified as indirect outcomes, most directors reported 
improved self-evaluation of the academic program, improved faculty 
attitude toward assessment activities, and greater visibility of assessment 
activities. 



When asked to describe other significant outcomes, four directors reported a 
significant amount of "self-improvement" in their personal understanding or 
student outcomes assessment, their ability to teach, and clarification of their 




program's objectives. Two also referred to simiiar improvements enjoyea bv ail 
faculty participating in the project. Two directors reported that the project had 
resulted in changes/improvements in their programs' curriculum and definition 
of objectives. One director reported that the project formed the basis for a 
prototype assessment procedure that is being piloted for use statewide. 
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Conclusions and 
Recommendations 

The conclusions and recommendations drawn from an overview of the study's 
results are organized in terms of the four major variables in association with 
project effectiveness: faculty participation, administrative support, backgrounds 
of project directors, and measurement adequacy. 

FACULTY INVOLVEMENT 

Recruiting and maintaining faculty support was a key variable in project 
success, but it also showed high variability across projects. One concern in ^ 
nearlv all faculty groups was "the intended primary use of the outcomes data, 
particularly where data suggested evidence of teaching/program effectiveness. 
Another might be described as the worry over the human capital costs ot 
department-level assessment activities. This turned up as a particular concern 
for junior faculty. Many junior faculty perceived that research on teaching and 
learning was regarded as "second-tier" research which might r, : be counted in 
the tenure/promotion process. Several project directors indicated that assess- 
ment activities favor two faculty groups: (1) those in social/behavioral disci- 
plines, and (2) those whose professional research activities "fit" with assess- 
ment research. 

Recruiting and maintaining faculty support was a key variable in project 



success 




Three general guidelines for establishing and maintaining faculty involvement 
in assessment can be drawn from the experiences of CSU project directors: 

1 . Educate participants about the value of assessing student outcomes. 

The motivation required to commit to outcomes assessment is 
dependent upon a general perception that this effort will have positive 
results. Informational meetings can be used to acquaint participants 
with the purposes of assessment, especially in terms of improving 
teaching and learning and academic programs. On an individual basis, 
participation will be enhanced if assessment activities are recognized 
in departmental faculty performance criteria. 

2. Maintain local control of the project. Support is dependent upon the 
perception of local autonomy. Local control in the planning, imple- 
mentation, and dissemination of the project is important to the faculty 
belief that the project will be relevant to and consistent with their own 
objectives. However, involvement of approprLte faculty governance 
groups is also of critical importance. 
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3. Overcome perceived threats. It is important that project goals be 
clearly stated and related to the educational program, and that faculty 
members participate in planning the uses of the data to be collected. It 
is also important to ensure that data are used for their intended pur- 
poses, and that proper safeguards are in place to prevent potential 
misuse of assessment data. 



ADMINISTRATIVE SUPPORT 

The importance of the development of administrative support was consistently 
reported across projects. Data from the majority of respondents suggested that 
"in-kind" resources, publicity, campus-level coordination, and establishing a 
climate receptive of assessment initiatives were important positive contribu- 
tions of administrative offices. Even in this positive environment, however, 
better integration of the assessment agenda in campus-level policy and more 
concrete recognition of assessment activities for professions development are 
needed. 

...better integration of the assessment agenda in campus-level policy and 
more concrete recognition of assessment activities for prof essonal develop- 
ment are needed. 



Two general recommendations can be made on the basis of this study's 
findings. The first deals with the establishment of administrative support, the 
second partially defines the nature of the support sought: 

4. Educate administrators about the value of assessing student out- 
comes. Just as executives of business organizations value economic 
indicators for their companies, educational administrators need to 
understand the potential value of performance feedback inherent in 
the assessment of student outcomes. They must also be sensitized to 
the need for discretionary, constructive use of such data. 

5. Obtain support of administrators for recognizing and rewarding the 
development of outcomes assessment in one's field as legitimate 
professional development. Outcomes assessment activities should be 
recognized as legitimate professional development by chairs, deans, 
and department/school/university evaluation committees in control of 
the retention, promotion, and tenure process. This recognition should 
be treated equally across disciplines. 

TRAINING/EXPERIENCE OF PROJECT DIRECTORS 

Project directors' training/experience in measurement and analysis was key to 
project effectiveness, and here there were important differences. Some project 
directors reported dismay over the difficulties in learning assessment proce- 
dures as the project progressed. A number of respondents echoed the senti- 
ments of one director who felt that the project lost momentum "just as experi- 
ence and proficiency began to develop." 
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These observations clearly suggest the importance of training in outcomes 
assessment for those administering assessment programs. Two general recom- 
mendations flow from the comments made by project leaders: 

6. Evaluate the project director's training/experience specific to assess- 
ing student outcomes. General knowledge and experience as an 
educator is not enough. Project directors need to have an adequate 
level of expertise in planning and implementing assessment programs 
as well as selecting or designing appropriate measuring instruments for 
data collection, analysis, and reporting. Directors' training in the field 
will offset any lack of experience of other project participants. 

7. Provide continuous opportunities for training in and exposure to 
outcomes assessment. Continuous interest and updated field knowl- 
edge of project directors are necessary for successful implementation 
and eventual institutionalization of assessment programs. Therefore, 
opportunities for professional development in this area, such as 
attending national assessment conferences, receiving training from 
experienced practitioners, or sharing information among colleagues, 
are very important for kindling directors' interest in the process. 

PROJECT PROCEDURES AND MEASUREMENT ADEQUACY 

The key methodological variable was the development or adoption of adequate 
measures of student outcomes. Adequacy of measurement implies several 
aspects. The first is the simple psychometric properties of the assessment. This 
involves the reliability and validity of the assessment procedures. As an ex- 
ample of awareness of measurement adequacy, several directors did an 
excellent job of assessing the inter-rater reliability of judges producing qualita- 
tive assessments of student projects or papers. Others spent considerable time 
and consulted widely with their peers to evaluate the content validity of their 
assessments. This process often has positive, retroactive impact on curriculum 
and teaching strategies. 

Multicultural sensitivity also contributes to the adequacy of measurement, 
especially in reference to the inferences drawn from assessment scores. A 
critical question for directors to ask is: "What assessment procedures will 
provide all students with an equitable opportunity to demonstrate their compe- 
tence?" This consideration should result in the production of multiple, more 
creative indices that wouid provide a more comprehensive picture of student 
achievement. 



The bottom line is that measures cannot be hapltazardly developed or selected. 



Multiple types of assessment also enable directors to obtain feedback on more 
than one type of outcomes. Rather than focusing solely on content or 
cognitivelv based outcomes, additional measures of affective and attitudinal 
variables should result in a much richer, more complex basis for judgments of 
program adequacy. 
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The bottom line Is that measures cannot be haphazardly developed or selected. 
Future faculty initiators of student outcomes assessment will decide for them- 
selves what and when to assess, but the findings of this study indicate that they 
will need a good deal of technical support in order to implement an effective 
assessment program. 

The following summary recommendations are derived from observations 
related to procedural and measurement adequacy: 

8. Clearly define educational objectives* The nature of the assessment 
tools cannot be determined until the desired outcomes are described, 
A healthy re-evaluation of curriculum and program goals is a 
necessary precursor to the development of the actual assessment 
instruments. 

9. Use multiple measurement techiques. Since educational objectives 
are seldom unidimensional, it makes little sense to attempt to assess 
educational criteria with a single measure. The richest data sources 
enabling the clearest assessment of program outcomes involve 
combinations of contents tests, performance-based demonstrations, 
attitude assessments, affective measures, etc. 

1 0. Be sensitive to test fairness across constituent groups. Effective 
assessment programs must address issues of differential performance 
across groups. Assessment procedures must take into account and 
minimize possible test bias resulting from such cultural/socioeconomic 
factors as native language speaking and disadvantaged preparation for 
higher education. 

1 1 . Examine measurement reliability and validity for all instruments used. 

The importance of this recommendation cannot be overemphasized. If 
an assessment instrument does not possess adequate psychometric 
properties, it provides no basis for meaningful inference concerning 
the relative performance of the student or the success of the academic 
program. Assessment project directors should work together with 
psychometrics experts, if they themselves are not, to validate the 
measures to be used before starting the data collection procedure. 

1 2. Plan to adequately disseminate project results. Projects with explicit 
dissemination plans will keep project directors active in publishing 
and reporting project results. Stories of successful projects and their 
results are helpful to other faculty-initiated assessment programs, and 
help to build the knowledge base for future assessment efforts. 
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ACADEMIC PROGRAM IMPROVEMENT 



The Academic Program Improvement (API) Campus Grants Program was 
created to assist faculty in developing innovative approaches to improve both 
teaching and student learning through pilot projects, workshops, and confer- 
ences. From 1972 to 1992, an estimated 150,000 students and 13,000 faculty 
directly participated in one of the projects, workshops, or conferences spon- 
sored by the program. An even greater number have benefited indirectly as the 
effects of instructional improvements are felt by subsequent groups of students 
and as faculty learn from colleagues who have adopted these changes in 
curricula and teaching methods. 

Funds have been provided for faculty assigned time, clerical and student 
assistance, and supplies and materials not otherwise available through normal 
campus and departmental support. The overwhelming majority of the projects 
it.;tially supported through API continue with support from the campus institu- 
tional budget. 

For additional information on the API program, or on any of the projects 
described in this brochure, the reader should contact: 



Institute for Teaching and Learning 

Office of the Chancellor 

The California State University 

400 Golden Shore 

Long Beach, California 90802-4275 

(310) 985-2607 
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